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1.  INTRODUCTION 


A  vertex  in  a  directed  graph  is  in  a  knot  if  for  every 
vertex  v^  reachable  from  v^ ,  v^  is  reachable  from  v^  .  Chang  [  1  ] 
shows  that  knot  is  a  useful  concept  in  deadlock  detection. 

Dijkstra  [  2  ]  has  proposed  a  distributed  algorithm  for  detecting 
if  a  given  process  in  a  network  of  processes  is  in  a  knot. 

His  algorithm  is  based  on  his  previous  work  with  C.  S.  Scholten 
[  3  ]  on  termination  detection  of  diffusing  computations.  We 
propose  an  algorithm  for  knot  detection  which  is  also  based 
on  [3],  but  is  conceptually  simpler.  We  also  discuss  the 
extensions  of  our  algorithm  to  a  more  general  class  of  problems. 

2.  MODEL  OF  A  NETWORK  OF  COMMUNICATING  PROCESSES 

A  process  is  a  sequential  program  which  can  communicate 
with  other  processes  by  sending/receiving  messages.  Two  pro¬ 
cesses  P  and  Q  are  said  to  be  neighbours  if  they  can  communi¬ 
cate  directly  with  one  another  without  having  messages  go 
through  intermediate  processes.  We  assume  that  communication 
channels  are  bi-directional:  if  P  can  send  messages  to  Q  then 
Q  can  send  messages  to  P.  A  process  knows  its  neighbours  but  is 
otherwise  ignorant  of  the  gereral  communication  structure  of  the  network . 

We  assume  a  very  simple  protocol  for  message  communication; 
this  protocol  is  equivalent  to  the  one  used  by  Dijkstra  and 
Scholten  13].  Every  process  has  an  input  buffer  of  unbounded 
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length.  If  process  P  sends  a  message  to  a  neighbour  process 
Q,  then  the  message  gets  appended  at  the  end  of  the  input 
buffer  of  Q  after  a  finite,  arbitrary  delay.  We  assume  that 
(1)  messages  are  not  lost  or  altered  during  transmission,  (2) 
messages  sent  from  P  to  Q  arrive  at  Q's  input  buffer  in  the 
order  sent,  and  (3)  two  messages  arriving  simultaneously  at 
an  input  buffer  are  ordered  arbitrarily  and  appended  to  the 
buffer.  A  process  receives  a  message  by  removing  it  from  its 
input  buffer. 

The  assumption  of  unbounded  length  buffers  is  for  ease  of 
exposition.  We  show,  in  section  5.1,  that  the  input  buffer 
length  of  process  Q  can  be  bounded  by  the  number  of  neighbours 
of  Q. 


3.  A  DISTRIBUTED  ALGORITHM  FOR  KNOT  DETECTION 

Consider  a  network  of  processes  corresponding  to  a  given 
directed  graph  G;  there  is  a  one-to-one  correspondence  between 
processes  in  the  network  and  vertices  in  the  graph  and  a 
process  p^  in  the  network  represents  vertex  v^  in  G,  for  all 
i,  and  P^»Pj  are  neighbours  if  edge  (v^,Vj)  or  (Vj,v^)  exists 
in  G.  Process  p^  initiates  a  computation  to  determine  if 
v^  is  in  a  knot. 

3.1  Local  Variables  of  Processes 

Every  process  p^  maintains  the  following  variables . 
succeeding (i)  :  this  boolean  variable  is  set  true  when  p^ 

determines  that  v^  is  reachable  from  Vj^. 


pccial 


□  □ 
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Initially  this  variable  is  false  for  all  p^  , 
i  ^  1  and  is  true  for  p^.  Eventually  succeeding (i ) 
will  be  true  if  and  only  if  v^  is  reachable 
from  v^. 

preceding (i)  :  Same  as  above  except  that  it  represents  whether 

v^  is  reachable  from  v^ . 

subordinated):  this  is  integer  valued  and  will  be  set  to  1  if 

and  only  if  succeeding (i )  and  not  preceding  d); 
else  it  will  be  set  to  0.  v^  is  in  a  knot  if 
and  only  if  subordinate (i )  is  eventually  zero 
for  every  process  i. 

cs(i)  :  this  is  an  integer  valued  variable,  which 

keeps  the  partial  sum  of  some  subordinate 
variables.  A  goal  of  the  program  is  to  estab¬ 
lish  the  following  at  termination: 

cs(l)  =  £  subordinate (i) 

i 

Therefore  v^  is  in  a  knot  if  and  only  if 
cs(l)  =  0  at  termination. 

We  discuss  in  section  3.2  the  different  types  of  messages 
sent  among  processes.  In  short,  a  process  p^  may  send  a  message 
to  Pj  and  Pj  sends  an  acknowledgement  (ack)  to  p^  for  every 
message  that  p^  receives  from  p^.  We  introduce  the  following 
variables  related  to  message  and  ack  transmission. 


num  (i ) 


:  is  the  number  of  unacknowledged  messages,  i.e. 


the  number  of  messages  sent  by  this  process  for 
which  acks  have  not  been  received  so  far. 
father (i)  :  is  a  process  from  which  p^,  i^l,  received  a 

message  when  its  num(i)  was  last  zero.  father (i) 
is  undefined  initially. 


Our  goal  is  to  maintain  a  rooted  tree  structure  at  all 
times  over  processes  whose  num  >  0;  father  will  denote  the 
parent  in  this  tree  structure  and  p^  the  root. 

3.2  Messages  Sent  Among  Processes 

There  are  two  types  of  messages  sent  between  neighbours 
in  thi.'  algorithm. 

(i)  Structure  message  or  message  :  has  2  components 

(type,  p)  where,  type  =  sue  or  p re,  and 

p  is  the  identity  of  the  sender  process.  Process  p^  sends 
(sue,  p^  to  Pj  if  there  is  a  path  from  v-^  to  v^  in  which  v^^ 
is  the  prefinal  vertex.  Process  p^  sends  (pre,  p^)  to  p^ 
if  there  is  a  path  from  v..  to  in  which  follows  in 
the  path. 

(ii)  Acknowledgement  message  or  ack:  is  of  the  form  (ack,  c) , 
where  c  is  an  integer.  Acks  are  used  to  update  cs  and  num. 

The  entire  computation  terminates  when  process  p^  receives 
acks  for  all  messages  that  it  sent;  i.e.  when  mun(l)  is 
decremented  to  zero.  Acks  for  all  messages  are  sent  back 
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as  scon  as  the  messages  are  received  except  for  messages 
received  from  father;  an  ack  to  a  father  is  sent  only  when 
num  next  becomes  zero. 

Convention 

It  is  convenient  for  purposes  of  proof  to  define  an 
atomic  action  within  which  invariant  assertions  may  be  tem¬ 
porarily  violated  and  outside  which  the  invariants  must  hold. 

We  write  <A,;A0;...  A  >  to  show  that  executions  of  statements 
l  i  n 

Al'A2'***,An  must  ke  considered  as  an  atomic  action.  We  use 
Pascal  like  notation  with  the  added  commands  send  and  receive 
to  write  our  programs. 

3. 3  Knot  Detection  Algorithm 
Convention 

We  write  succeeding,  preceding,  etc.  for  succeeding (i ) , 
preceding (i ) ,  when  the  context  is  clear. 

Overview  of  the  Algorithm 

As  stated  earlier,  one  goal  of  the  algorithm  is  to  maintain 
a  rooted  directed  tree  structure  over  the  set  of  processes  p^ 
whose  num(i)  >  0.  The  root  of  the  tree  will  be  p^  and  father (i) 
will  be  the  parent  in  the  tree  for  p^,  i^l.  In  order  to  maintain 
the  tree  structure,  we  must  ensure  that,  (1)  a  process  p^ ,  ij*l, 
acquires  a  father  only  if  it  does  not  have  ohe  currently:  this 
is  guaranteed  since  a  process  acquires  a  father  only  when  its 
num(i)  becomes  nonzero,  and  (2)  a  process  p^  can  be  removed  from 
the  tree,  i.e.  set  its  num(i)  =0,  only  if  it  was  a  leaf  node: 
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this  will  be  guaranteed  by  every  process  sending  its  last  ack 
to  its  father.  Computation  terminates  when  the  tree  is  empty. 

We  will  also  maintain  the  invariant  (1)  given  in  lemma 
4.2,  which  states  that  the  sum  of  cs  over  all  processes  plus 
those  in  the  acks  in  transit  equal  the  sum  of  subordinates 
over  all  processes.  The  algorithm  will  ensure  that  if  num(i)  =  0 
and  i^l,  then  cs(i)  =0.  Therefore,  when  the  tree  is  empty, 
cs(i)  =  0,  for  all  i,  i/1  and  hence 

cs  ( 1 )  =  I  subordinated), 
i 

Process  p^  is  in  a  knot  if  and  only  if  cs(l)  =  0. 


3.3.1  Algorithm  for 


Initialization 

begin 

father  is  undefined; 

subordinate  :=  0;  cs  :=  0;  num  :=  0; 
<succeeding  :=  true ; 

num  :=  num  +  number  of  successors  of  v^; 
send (sue,  p.)  to  all  successors>; 
<preceding  T=  true ; 

num  :=  num  +  number  of  predecessors  of  v^; 
send (pre ,  p.)  to  all  predecessors> 

end 


Upon  receiving  a  structure  message  (type,  p) 

send  (ack,  0)  to  p  (Ml) 


Upon  receiving  an  acknowledgement  (ack,  c) 
begin 

cs  :=  cs  +■  c;  num  :=  num  -  1;  (M2) 

if  num  =  0  then  terminate  computation  j 

(v.  is  in  a  knot  if  cs  =  0}  f 

end  i 
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3.3.2  Algorithm  for  ,  i  f  1 


Initialization 


begin 

father  is  undefined;  subordinate  :=  0;  cs  :=  0,  num  :=  0; 
succeeding  :=  false;  preceding  :=  false 

end 


Upon  receiving  a  message  (type,  p) 
begin 

{update  father  or  send  an  ack  immediately} 
if  num  =  0 

then  father  :=  p 

else  begin  <send  (ack,  cs)  to  p;  cs  :=  0>  end; 

{update  succeeding  and  preceding  if  necessary} 

if  type  =  sue  and  not  succeeding  {For  the  first  time, 

has  determined  that  is  reachable  from  v^} 

then 

begin  succeeding  :=  true; 

num  :=  num  +  number  of  successors  of  v.^; 
send  (sue,  p.)  to  all  successors>  1 

end;  1 

if  type  =  pre  and  not  preceding  {For  the  first  time,  p^ 

has  determined  that  v.  is  reachable  from  v. } 

1  i 

then 

Eegin  <preceding  ;=  true; 

num  :=  num  +  number  of  predecessors  of  v. ; 
send  (pre,  p.)  to  all  predecessors* 
end;  1 


{update  subordinate  if  necessary.  Also  update  cs  to  maintain 
the  invariant  in  lemma  4.2} 
if  succeeding  and  not  preceding 

subordinate  :=  1>  end 
subordinate  :=  0>  end; 

{send  ack  to  father  if  num  =0} 
if  num  =  0 

then  begin  send  (ack,  cs)  to  father;  cs  :=  0>  end 

end 


Eegin  <cs  :=  cs  -  subordinate  +  1; 
else 

Eegin  <cs  :=  cs  -  subordinate  +  0; 


(LI) 


(L2) 

(L3) 

(L4) 
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Upon  Receiving  an  acknowledgement  (ack,  c) 

begin 

cs  :=  cs  +  c;  num  :=  num  -  1;  (L5) 

if  num  =  0 

then 

begin  <send  (ack,  cs)  to  father;  cs  :=  0>  end  (L6) 

end 

4.  PROOF  OF  CORRECTNESS 

4. 1  Lemma 

At  any  point  in  the  computation,  the  set  of  processes 
with  num  >  0  form  a  rooted  tree  with  p^  as  the  root  and  the 
parent  relation  specified  by  the  local  variable  "father." 

Proof 

The  lemma  holds  vacuously  initially.  num(i)  and  father (i) 
may  be  changed  only  upon  receipt  of  a  message  or  an  ack  by 
process  i.  If  a  process  with  num  >  0  receives  a  message 
then  it  does  not  alter  its  father,  thus  preserving  the  tree 
property.  Similarly,  if  a  process  has  num  >  0  after  processing 
an  ack,  it  does  not  alter  the  tree  structure.  If  a  process  p^ 
changes  num(j)  from  zero  then  it  must  have  received  a  message 
from  some  other  process  p^  on  the  tree  and  must  have  set 
father (j)  =  i,  thus  preserving  the  tree  property. 

We  now  show  that  only  a  leaf  node  can  decrement  its  num 
to  zero.  If  p^  is  on  the  tree  and  is  not  a  leaf  then  there 
is  a  process  with  num(j)  >  0  and  father(j)  =  i;  then  p^ 
will  not  return  an  ack  to  p^  while  p..  remains  on  the  tree  and 
hence  num(i)  >  0,  while  p^  remains  on  the  tree.  Therefore 
only  a  leaf  node  can  decrement  its  num  to  0,  which  preserves 
the  tree  property. 
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Let  T,  at  any  point  in  computation,  denote  the  set  of 
ack  messages  which  are  in  Transit,  i.e.  which  have  been  sent 
but  have  not  yet  been  received. 

4 . 2  Lemma 

The  following  is  an  invariant. 

I  cs(i)  +  £  c  =  I  subordinate  (i )  (1) 

i  (ack,c)eT  i 

Proof 

The  lemma  holds  initially  since  all  the  terms  in  the 
equation  are  zero.  For  p^ ,  i  /  1,  the  terms  in  the  equations 
are  modified  only  at  program  points  LI  through  L6 ,  and  for  p^, 
these  terms  can  be  modified  only  at  Ml  or  M2.  The  reader  may 
easily  convince  himself  that  the  equation  is  left  invariant 
by  the  execution  of  the  statements  at  these  program  points. 

4.  3  Theorem 

Assume  that  process  p^  terminates  computation  (in  step  M2) . 
cs(l)  =  0  if  and  only  if  v-^  ia  in  a  knot. 

Proof 

We  will  first  show  that  when  p^  terminates  computation 
(I)  cs(i)  *  0  for  i  /  1,  and  (El)  subordinate  (i )  is  correctly 
set  and  (III) the  set  T  is  empty.  The  theorem  follows  directly 
from  the  invariant  proven  in  lemma  4.2. 

(I)  When  p^  terminates  computation  in  step  M2,  num(l)  =  0. 

Then  the  tree  is  empty  since  p^  was  the  root  of  the  tree. 
Therefore  num(i)  =  0  for  all  i.  If  num(i)  =  0  then  cs(i)  =  0, 
for  all  i,  i/1,  because  every  change  to  num(i)  is  followed  by 
the  code  to  set  cs(i)  to  0  if  num(i)  is  0  (steps  L4,L6). 
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(II)  If  v^  is  reachable  from  v^  ,  it  follows  by  induction  on 
path  length  to  that  will  eventually  receive  a  message 
which  will  result  in  succeeding (i )  set  true ;  succeeding (i ) 
remains  true  thereafter.  Similarly  for  preceding  (i ) .  There¬ 
fore  subordinate (i )  will  eventually  be  set  to  its  correct 
value.  When  assignment  is  made  to  succeeding  (i )  or  preceding (i ) , 

Pi  has  not  returned  an  ack  to  its  father  and  hence  the  compu¬ 
tation  could  not  be  over.  Therefore  these  variables  are 
assigned  their  correct  values  before  the  termination  of  computation. 

(III)  Since  the  tree  is  empty,  every  process  must  have  received 

acks  corresponding  to  all  messages  sent.  Therefore  there  can  be 
no  ack  in  transit,  i.e.  set  T  is  empty. 

4.4  Lemma 

Pj^  will  terminate  computation  in  finite  time. 

Proof 

A  process  p^  sends  at  most  two  messages  (type,  pi), 
to  any  other  process  p^  because  (1)  a  message  is  sent  only 
when  succeeding  or  preceding  is  set  to  true  and  (2)  succeeding 
and  preceding  are  never  reset  to  false.  Because  the  graph  is 
finite  the  total  number  of  messages  sent  is  bounded.  Hence 
the  total  number  of  acks  sent  is  also  bounded.  Observe  that 
every  process  must  send  or  receive  either  a  message  or  an  ack 
every  time  it  starts  to  execute.  Therefore  a  process  can 
switch  from  idle  to  executing  only  a  finite  number  of  times. 

There  are  no  loops  in  the  program;  therefore  every  executing 
process  will  become  idle  in  finite  time.  Hence  every  process 
in  the  network  will  cease  to  execute  in  finite  time  and  no 
more  messages  or  acks  will  be  sent  or  received  from  then  on. 


11 


We  now  show  that  the  tree  must  be  empty  at  this  point. 

If  not,  let  p^  be  a  leaf  node  of  the  tree;  num(i)  >  0  since 
p^  is  on  the  tree.  There  is  no  p^  on  the  tree  for  which 
father  (j)  =  p^  and  hence  p^  must  have  received  all  its  out¬ 
standing  acks;  therefore  num(i)  =  0!  Contradiction! 

5.  NOTES  ON  THE  KNOT  DETECTION  ALGORITHM 

5. 1  Bounding  the  Buffer  Size 

We  assumed  earlier  for  purposes  of  exposition  that  buffers 
are  of  unbounded  length.  In  the  knot  detection  algorithm  a 
process  sends  at  most  2  messages  to  any  neighbour  process  and 
therefore  no  process  sends  more  than  2  acks  to  any  other  pro¬ 
cess.  Hence  the  buffer  length  for  any  process  need  not  exceed 
4  times  the  number  of  neighbours  of  the  process. 

5 . 2  Efficiency 

This  algorithm  is  superior  to  the  brute-force  algorithm 
in  which;  (1)  process  p^  computes  successor*,  the  set  of  ver¬ 
tices  reachable  from  v^  and  (2)  predecessor*,  the  set  of 
vertices  that  can  reach  v-^  and  (3)  then  declares  that  v^  is 
in  a  knot  if  and  only  if  successor*  £  predecessor*.  The  compu¬ 
tation  of  successor*  (predecessor*)  can  be  done  by  using  an 
algorithm  similar  to  the  one  proposed  here  -  every  ack  carries 
with  it  a  set  of  successors  (predecessors) .  Therefore  a  suc¬ 
cessor  at  distance  d  from  v^,  will  have  its  identity  transmitted 
through  d  processes  to  reach  v^.  Total  message  length  will  be 
at  least  0 (N  ),  for  an  N-vertex  graph  as  opposed  to  0(E)  for 
our  algorithm  where  E  is  the  number  of  edges. 
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6.  EXTENSIONS 

We  show  in  this  section  that  the  ideas  in  the  knot  detec¬ 
tion  algorithm  can  be  extended  to  solve  a  very  general  class 
of  problems.  Consider  a  distributed  computation  which  is 
initiated  by  process  p^  sending  messages  to  some  of  its  neigh¬ 
bors.  Any  other  process  can  send  messages  only  after  receiving 
a  message.  The  computation  terminates  when  no  process  has 
any  more  messages  to  send  and  all  messages  that  have  been  sent 
have  been  received.  Dijkstra  and  Scholten  [3]  were  the  first 
to  identify  this  class  of  computations,  which  they  call  diffusing 
computations .  They  proposed  an  algorithm,  using  the  growing 
and  shrinking  tree,  to  detect  termination  of  diffusing  computations. 
Our  contribution  is  to  show  how  the  same  idea  may  be  exploited 
to  compute  a  network-wide  function  of  locally  computed  results. 

Let  local- result (i)  denote  some  computed  result  at  process 
p^ ,  at  termination  of  the  entire  computation.  It  is  required 
to  compute  global-result  at  the  termination  of  computation, 
where 

global-result  =  /(local-result (i) ,  for  all  i)  (2) 
where  f  is  any  arbitrary  computable  function. 

The  knot  detection  algorithm  computed  the  global  result 
cs  (1)  , 

cs(l)  =  1  subordinate (i) ,  (3) 

i 


i.e.  f  =  l 


We  propose  two  schemes  to  compute  network-wide  functions. 


Note  that  our  algorithm  can  bo  used  to  develop  distributed 
algorithms  according  to  the  following  methodology:  in  order  to 
compute  some  global-result,  invent  a  function  f  and  local-result (i) 
satisfying  (1)  and  then  design  a  distributed  algorithm  to  com¬ 
pute  local-result (i)  at  process  p^,  for  all  i.  Then  superimpose 
our  algorithm  to  compute  the  global-result.  A  variation  of 
this  idea  appears  in  [4],  where  a  number  of  other  problems 
amenable  to  this  approach,  are  listed. 


k 
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One  difficulty  with  a  straightforward  implementation  is 
that  a  process  cannot  know  when  network  computation  has  termi¬ 
nated.  Process  p^  knows  that  network  computation  can  terminate 
only  when  num{i)  =  0;  however,  cannot  assert  the  converse, 
i.e.  that  network  computation  may  not  have  terminated  even  if 
num(i)  =  0.  Hence  p^  must  send  back  its  current  value  of 
local-result (i)  to  its  father  every  time  that  it  decrements 

num(i)  to  zero.  This  causes  a  problem:  p  may  send  back  a 

i 

local-result  to  its  father,  and  subsequently  get  another  message 
which  causes  it  to  compute  a  new  local-result.  Therefore  p^ 
must  cancel  the  old  local-result  value.  We  propose  two 
mechanisms  for  cancelling  out-of-date  local  results:  bags  and 
time- stamps. 

To  simplify  exposition  in  our  discussion  of  cancellation 
schemes  we  will  assume  that  there  is  no  delay  between  sending 
and  receiving  a  message,  i.e.  there  is  never  any  message 
in  transit:  the  reader  can  easily  convince  himself  that  the 
arguments  also  apply  when  the  transmission  delay  is  not  zero. 

6 . 1  Bags 

Each  process  p^  maintains  two  bags  all(i)  and  cancelled (i) 
Each  bag  element  is  of  the  form  (j,  local-result (j )) .  If 
(j,x)  is  an  element  in  cancelled (i)  then  process  p^  has 
definitely  cancelled  an  out-of-date  local-result  x.  If  (j,x) 
is  an  element  of  all(i),  then  at  sometime  p..  posted  a  local 
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result  x.  The  elements  in  all(i)  are  not  necessarily  current. 
Every  local  result  that  p^  has  posted  appears  in  the  union 
of  bags  all(i),  for  every  i.  Similarly,  all  local  results 
that  Pj  has  cancelled  appear  in  the  union  of  cancelled  (i ) , 
for  every  i.  Therefore  p^'s  current  local  result  is  in  the 
difference  of  these  two  bag  unions.  In  other  words,  the  goal 
is  to  maintain  the  following  invariant.  Let  r(j)  denote  the 
current  local  result  of  process  j ,  and  let  U  denote  the  union 
operation  over  bags. 

V  (j,r(j)>  =  U  all(i)  -  U  cancelled  (i) 

j  i  i 

Initially,  all(i)  holds  the  initial  local  result  of  p.^ 
and  cancelled (i)  is  empty.  To  post  a  current  local  result 
x  and  cancel  the  previous  local  result  y,  process  p^  adds 
(i,x)  to  all(i)  and  (i,y)  to  cancelled (i) . 

Two  bags  a  bag  and  c  bag  are  returned  with  every  ack  in 
the  form  (ack,  a  bag,  c  bag  ) .  When  p^  sends  an  ack  it  takes 
the  elements  out  of  bag  all(j)  and  puts  them  into  a  bag,  and 
similarly  puts  elements  from  cancelled (j)  into  cbag,  and  then 
sends  a  bag  and  cbag  along  with  the  ack.  If  p^  receives  (ack, 
a  bag,  cbag)  it  adds  the  contents  of  a  bag  to  all(i)  and  cbag 
to  cancelled (i) . 

At  termination,  all(i)  and  cancelled (i)  will  be  empty 
for  i  f  1,  and  cancelled (1)  will  contain  tuples  corresponding 
to  all  cancelled  local-results,  and  all(l)  will  contain 
tuples  corresponding  to  all  local-results,  current  and 


cancelled.  By  removing  the  cancelled  results  (i.e.  elements 
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of  cancelled (1) )  from  all(l),  p^  can  determine  the  current 
local-results  for  all  processes.  The  knot  detection  algorithm 
of  section  3  uses  the  bag  idea;  the  information  in  the  two 
bags  have  been  condensed  into  a  single  integer  cs .  Adding  an 
element  (j,x)  to  all(i)  is  implemented  by  incrementing  cs(i) 
by  x.  Adding  an  element  (j,y)  to  cancelled (i)  is  achieved  by 
decrementing  cs(i)  by  y. 

A  Note  on  Efficiency 

The  sizes  of  the  bags  returned  with  acks  can  be  reduced 
by  having  each  process  remove  all  elements  common  to  all(i) 
and  cancelled (i)  from  both  all(i)  and  cancelled (i ) . 

6 . 2  Time- Stamps 

Each  process  p^  maintains  a  set  S(i)  of  triples  of  the 
form  (j,  n(j),  local-result (j ) )  where  n(j)  is  a  time-stamp 
local  to  process  p .  When  a  process  p^  wishes  to  post  a  new 
local-result  x  (and  cancel  an  out-of-date  result)  it  increments 
n(i)  and  adds  (i,  n(i),  x)  to  S. 

When  p^  sends  an  ack,  it  sends  (ack,  S(i)),  and  then  sets 
S(i)  to  empty.  Upon  receiving  an  ack,  (ack,  B) ,  p^  sets  S(i) 
to  the  union  of  S(i)  and  B.  Upon  termination,  S(i)  will  be 
empty  for  all  i  f  1,  and  S(l)  will  contain  all  tuples 
(i,  n(i),  S (i ) )  that  have  been  sent,  p^  can  identify  the 
current  local-results  because  they  will  be  associated  with 
the  latest  time-stamps. 

Efficiency 

The  sizes  of  the  sets  returned  with  acks  can  be  reduced 
by  having  each  process  discard  all  elements  in  S(i)  that 
it  can  identify  as  being  out-of-date. 
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